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ABSTRACT 


The  primary  objective  of  test  and  evaluation  in  the  Department  of 
Defense  is  to  support  the  acquisition  of  reliable  and  effective  weapon 
systems  for  the  operating  forces.  In  accomplishing  this  objective,  a 
methodology  is  required  that  builds  on  close  coupling  of  the  various 
testing  activities  and  emerging  technology.  This  paper  describes  these 
relationships  and  develops  alternative  system  concepts  that  lend  themselves 
to  inproving  test  and  evaluation  through  utilization  of  new  technology 
and  techniques  to  transfer  essential  information  in  a  more  effective 
manner. 

INITODOCTION 

Although  Test  and  Evaluation  (T&E)  are  firmly  established  as  an 
integral  part  of  the  weapon  system  acquisition  process  in  the  Department 
of  Defense  (DOD) ,  significant  problems  have  historically  inhibited  its 
effectiveness. 

Visibility  is  often  given  to  cost,  schedule,  and  performance  problems 
associated  with  test  and  evaluation  of  weapon  systems.  However,  the 
root  causes  of  these  problems  have  traditionally  been  overlooked  in  periodic 
sweeping  reforms.  In  attempting  to  identify  these  causes,  this  writer 
did  a  survey  and  analysis  of  severed  major  weapon  system  program  assessments 
and  associated  decision  guidance  issued  at  key  production  decision  points 
during  the  past  three  years.  Dominant  problems  with  cannon  elements 
distributed  across  all  programs  were  identified  that  required  extensive 
corrective  actions.  A  summary  listing  is  presented  below: 


o  Test  planning  was  not  timely  and  frequently  objectives,  goals 
and  thresholds  were  either  not  well  defined  or  not  met  during  Operational 
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Test  and  Evaluation. 

o  Due  to  late  identification  of  major  deficiencies,  extensive  Follow- 
on  Test  and  Evaluation  (FCT&E)  was  required  to  verify  mission  effectiveness 
and  operational  suitability.  This  resulted  in  conditional  decisions  at 
higher  risk  in  approximately  74%  of  the  programs  reviewed.  Reliability, 
maintenance,  human  factors,  training,  logistics,  and  software  immaturity 
were  dominant  deficiencies  identified  in  CT&E  assessment  reports. 

o  Considerable  blurring  of  Developmental  Testing  and  Operational 
Testing  resulted  in  production  decisions  on  hardware  and  software  that 
were  not  representative  of  systems  to  be  fielded;  therefore,  support  was 
highly  questionable. 

o  Inadequate  resources  were  available  for  critical  production  transition 
periods  due  to  limitations  on  reprogramming  or  built-in  budget  constraints. 

Although  test  and  evaluation  are  the  common  element  in  many  of  the 
above  problems,  this  was  not  the  root  cause  of  these  problems.  A  more 
indepth  analysis  coupled  with  personal  participation  in  the  process  at 
both  the  Service  and  OSD- levels,  support  the  conclusion  that  inadequate 
planning  and  information  flow  were  major  contributors  to  these  problems. 
Organizational  structures  and  compressed  schedules  inhibited  timely  and 
meaningful  communications  between  program  management  offices  and  the 
various  hierarchical  management  structures  within  the  defense  establishment. 
This  resulted  in  inefficient  resource  allocations,  schedule  restrictions, 
critical  transition  deficiencies  and  higher  decision  risks.  The  result 
of  this  dilemma  was  lack  of  quality  T&E  information  (vice  quantity)  that 
defined  in  advance  of  essential  decision  points  each  weapon  system's 
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operational  effectiveness  and  suitability.  Although  major  program  review 
points  were  well  known,  the  essential  program  objectives  and  supporting 
data  required  for  executive  level  decisions  were  not  clearly  quantified 
in  advance.  Therefore,  resource  adjustments  were  not  accomplished  in  a 
timely  manner  and  most  decisions  rendered  were  conditional  requiring 
extensive  after-the-fact  corrective  actions.  Essential  to  correction  of 
this  situation  are  early  identification  of  critical  test  issues  and  quality 
horizontal  data  flows  that  closely  couple  critical  events,  resource 
allocations  and  reviews.  Such  an  effort  requires  a  more  effective 
collection  of  data  and  more  timely  processing  that  accanmodates  the 
cognitive  styles  and  judgemental  heuristics  of  higher  level  decision 
makers.  A  detailed  functional  information  flow  with  essential  data 
elements  identified  must  be  developed  as  a  first  order  of  business.  The 
results  of  this  effort  should  provide  order  and  structure  necessary  for 
subsequent  development  of  a  decision  support  system  that  could  meet  the 
increasing  demands  being  placed  on  Defense  test  and  evaluation. 


TOE  DECISION  PROCESS 

The  decision  process  in  the  Office  of  the  Secretary  of  Defense  (OSD) 
utilizes  highly  structured  procedures  supported  by  many  unstructured 
processes  with  primary  inputs  from  the  Defense  Components .  Therefore, 
contrary  to  the  opinion  of  many  who  have  a  marginal  outside  perception 
or  limited  information,  there  are  many  variables  that  affect  the  effective¬ 
ness  of  these  processes. 

* 

Descriptive  Model  of  the  Decision  Process 

The  policy  and  control  procedures  for  major  acquisitions  are  specified 
for  reviews  at  critical  transition  milestones  by  various  hierarchical 


organizations  within  the  Department  of  Defense.  General  problem-solving 
strategies  are  often  used  to  deal  with  complex  problems  at  these  reviews 
with  analogy,  redefinition,  alternative  strategy,  quantitative  judgment 
and  intuition  among  the  many  processes  utilized  in  this  form  of  decision 
making.  Therefore,  the  cognitive  style  of  decision  makers  is  a  fundamental 
factor  influencing  results.  It  is  the  contention  of  this  author  that 
such  influence  is  significant  and  may  be  the  most  important  factor  in 
the  decision  process  at  the  highest  levels  in  the  Department  of  Defense. 

At  the  OSD  level,  assessments  by  various  staff  experts,  including  the 
Director  Defense  Test  and  Evaluation,  establish  alternatives  and  recommenda¬ 
tions  that  are  provided  to  each  member  of  a  Defense  Systems  Acquisition 
Review  Council  (DSARC)  prior  to  a  comprehensive  briefing  by  the  program 
manager.  Results  of  this  council's  deliberations  are  forwarded  to  the 
Deputy  Secretary  or  Secretary  of  Defense  for  incorporation  into  a  Decision 
Memorandum  directed  to  the  Service  responsible  for  implementation.  The 
traditional  textbook  models  that  appear  to  be  similar  to  this  process 
include  the  Anthony  Taxonomy  of  strategic  planning-management  control- 
operational  control,  the  "satisficing"  process-oriented  view,  and  the 
organization  procedures  view. 

Test  and  evaluation  often  contain  the  most  difficult  set  of  variables 
impacting  the  weapon  system  acquisition  decision  process;  therefore,  its 
functioned  relationships  form  the  core  of  the  process  description  in 
this  report.  A  draft  proposal  for  a  generic  model  of  Defense  Test  and 
Evaluation  is  provided  in  Appendix  1.  (The  reader  should  review  this 
appendix  before  proceeding.)  Sunntary  descriptions  of  policy,  processes, 
associative  functional  diagrams  and  data  forms  are  provided  as  the  primary 
elements  of  this  model,  with  essential  functions  directly  supporting  the 
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DSARC  process  being  items  4.3,  4.7,  4.8,  and  4.9.  The  most  important 
parts  of  this  model  are  the  logic  descriptions  and  data  sets  that  can  be 
utilized  to  establish  standards  for  T&E  processes  and  data  collection 
activities.  These  are  considered  to  be  the  essential  components  -that 
support  development  of  standards  necessary  for  a  T&E  distributed  data 
base  system.  Integration  of  these  and  associated  decision  support 
hierarchical  systems  are  shown  in  Figure  1. 

These  T&E  processes  and  associative  acquisition  relationships  are 
considered  to  be  basic  building  blocks  required  for  a  decision  support 
system  that  can  improve  existing  solutions  and  reduce  bounds  on  rationality, 
organization  roles  and  caimunication  channels.  Such  processes  should 
utilize  a  taxonomy  of  decision  rules  integrated  into  the  Bonczeks,  Holsapple, 
Whinston  generic  descriptions  for  decision  support.  A  functional  sample 
of  the  DSS  suggested  for  Service  Independent  Test  Activity  and  DDT&E  is 
shown  in  Figure  2.  Such  systems  would  be  computer-based  for  management 
decision  makers  who  are  deeding  with  semi structured  problems.  They  should 
provide  support  for  users  at  different  levels  in  all  phases  of  the  decision¬ 
making  process  on  a  hierarchical  relational  basis.  Particular  attention 
must  be  given  to  the  language  system  to  simplify  user  computation  and 
data  retrieval  but  not  limit  possible  artificial  intelligence  structuring 
techniques.  Obviously  these  descriptive  processes  require  considerably 
more  attention  to  detail  and  analysis;  however,  they  are  considered  to 
form  the  conceptual  framework  for  a  feasibility  study.  Suggested  steps 
for  this  feasibility  study  are  as  follows: 

o  Gather  facts  on  T&E  agencies  and  DDT&E 
o  identify  cross-Services  camonalities 
o  Validate  and  refine  Appendix  1  generic  model  of  T&E 


T&E  PROCESSES  AND  DECISION 
SUPPORT  RELATIONSHIPS 
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FUNCTIONAL  SAMPLE  DECISION 
SUPPORT  SYSTEM  (DSS) 


FIGURE  2 
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o  Develop  hierarchical  flow  charts 

o  Assess  the  feasibility  of  a  DSS  prototype  at  a  Service  Independent 
Test  Agency  or  Joint  Test  Activity 
o  Develop  an  implementation  plan 

Also  incorporated  into  this  study  must  be  key  variables  that  would 
influence  implementation.  Lucas  in  his  earlier  book  (1976)  on  "The 
Implementation  of  Computer-Based  Models"  lists  these  and  describes  their 
relationships.  His  later  book  (1981)  on  "Implementation:  The  Key  to 
Successful  Information  Systems"  gives  very  good  examples  on  application 
of  "Lewin  Force  Field"  analyses  and  a  comparison  of  design  activities 
with  critical  factors.  This  study  must  also  be  closely  coupled  with  the 
research  and  development  initiatives  described  in  the  tools  and  techniques 
for  T&E  section  of  this  report.  Key  to  effective  implementation  of  a 
T&E  automation  concept  is  connectivity  to  relational  data  bases  at  each 
of  the  Services'  Independent  Test  and  Evaluation  Agencies. 

Organizational  Support  Relationships 

As  indicated  in  Figure  3,  test  and  evaluation  has  been  firmly  integrated 
into  the  Department  of  Defense  management  structure.  However,  as  indicated 
in  the  introduction  of  this  report  processes  to  determine  weapon  systems 
operational  effectiveness  and  suitability  must  place  more  emphasis  on 
establishment  of  critical  test  issues  and  evaluation  criteria  during  the 
early  stages  of  the  acquisition  cycle.  This  requires  that  testing  methodology 
and  instrumentation  provide  a  mature  and  flexible  network  of  resources 
that  stress  weapon  systems  in  a  variety  of  environments.  Therefore, 
traditional  lines  of  demarcation  between  the  developer,  tester,  and  user 
must  be  modified  to  provide  positive  associative  relationships  that  increase 
the  potential  for  achieving  more  realistic  testing  conditions.  Such 
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close  coupling  of  test  activities  provides  conditions  favorable  for 
utilization  of  cannon  test  documentation  data  bases  as  programs  progress 
through  the  various  acquisition  phases.  In  support  of  this  concept, 
considerable  attention  must  be  given  to  software  verification  and  more 
attention  given  to  coordinated  utilization  of  simulation,  testbeds,  and 
prototypes  in  the  acquisition  process.  These  activities  linked  with 
technology  advances  during  the  next  several  years  are  expected  to  provide 
unique  opportunities  for  improving  test  and  evaluation,  while  at  the 
same  time  presenting  many  transition  challenges.  Essential  to  effective 
implementation  of  test  and  evaluation  policies  are  the  procedures  and 
organizational  support  provided  by  the  Services.  These  organizations 
must  provide  the  resources  and  commitment  to  principles  required  for  the 
Director  Defense  Test  and  Evaluation  to  effectively  support  major  program 
decision  milestones.  Basic  test  and  evaluation  principles  currently 
being  emphasized  to  accomplish  these  objectives  include: 

o  Initiation  of  Development  Test  and  Evaluation  (DT&E)  and 
Operational  Test  and  Evaluation  (OT&E)  early  to  identify  and  reduce  risks 
and  estimate  operational  potential. 

o  Establishing  critical  issues,  test  objectives  and  evaluation 
criteria  well  before  testing  begins  and  ensuring  that  these  are  related 
to  the  mission  need. 

o  Gathering  of  sufficient  test  data  with  appropriate  test  instrumenta¬ 
tion  to  minimize  subjective  judgements. 

o  Conduct  of  OT&E  by  Service  test  agencies  independent  of  the  developer 
and  user. 

Although  current  policy  states  that  successful  test  and  evaluation 
are  required  for  a  decision  to  commit  resources  or  advance  at  milestones 


established  for  key  acquisition  phases,  too  many  programs  are  now  requiring 
conditional  decisions.  This  has  forced  after-the-fact  adjustments  in 
programs  and  ccranitted  the  T&E  community  to  lengthy  schedules  that  seriously 
impact  limited  resources.  This  situation  is  ccmpunded  in  that  Service 
independent  test  agencies  are  also  beginning  to  get  actively  involved  in 
the  early  phases  of  the  acquisition  process.  Space  and  Cruise  Missile 
programs  are  examples  of  such  involvement  that  have  attempted  to  expedite 
acquisitions  by  combining  significant  phases  of  development  and  operational 
testing.  In  these  situations,  the  credibility  of  testing  must  be  maintained 
and  quality  independent  test  and  evaluation  reports  submitted  prior  to 
major  decision  milestones.  Such  procedures  are  essential  and  must  be 
included  in  test  planning  to  minimize  data  biases  that  distort  decisions. 
Planning  and  Assessment 

Planning  is  most  important  for  timely  establishment  of  procedures 
that  will  provide  an  effective  test  and  evaluation  program.  The  document 
that  is  utilized  to  describe  such  planning  in  DoD  is  the  Test  and  Evaluation 
Master  Plan  (TEMP) .  Contents  of  this  document  for  major  weapon  systems 
include: 

o  Mission  and  system  description 
o  Required  operational  and  technical  characteristics 
o  Critical  T&E  issues 
o  Management  responsibilities 
o  OT&E/OT&E  expected  or  actual  results 
o  Production  acceptance  T&E 

o  Test  articles  and  special  support  requirements. 

For  programs  requiring  Secretary  of  Defense  (SECDEF) ,  Deputy  Secretary 


of  Defense  (DEPSECDEF) ,  or  Under  Secretary  of  Defense  for  Research  and 


Engineering  (USDR&E)  decisions,  the  Director  Defense  Test  and  Evaluation 
(DDT&E)  must  approve  the  TEMPs  and  provide  an  assessment  of  the  weapon 
systems'  operational  effectiveness  and  suitability.  Assessments  require 
utilization  of  a  variety  of  analysis  techniques  with  TEMP's  forming  the 
primary  benchmark  criteria  for  test  and  evaluation  reporting  by  Service 
Independent  Test  Agencies  and  program  managers.  Such  comparison  of  measured 
results  from  operationally  realistic  tests  with  clearly  defined  requirements 
has  historically  proven  to  be  an  effective  means  of  quantifying  program 
risks.  A  simplified  listing  of  typical  weapon  system  elements  included 
in  T&E  is  presented  in  Figure  4. 

TOOLS  AND  TECHNIQUES  FOR  TEST  AND  EVALUATION 
Major  weapon  systems  in  the  Department  of  Defense  take  a  variety  of 
forms  with  designs  that  range  fran  mostly  software  systems  to  mostly 
hardware  systems.  Therefore,  developers  and  testers  of  weapon  systems 
utilize  a  multiple  of  techniques  to  validate  performance  of  subsystems 
and  systems  as  programs  progress  through  various  stages  of  acquisition. 

Recent  emphasis  on  compressed  schedules  and  greater  complexities  are 
forcing  more  attention  to  utilization  of  simulation  techniques  to  demonstrate 
performance  during  early  stages  of  weapon  system  developments,  since 
deployment  environments  are  difficult  to  duplicate,  almost  any  testing 
done  can  be  considered  some  form  of  simulation.  Such  simulations  can  be 
very  realistic,  as  in  operational  testing  of  production  systems,  or  they 
can  be  very  abstract,  as  in  development  testing  of  computer  models. 


GENERIC  DEFENSE  WEAPON  SYSTEMS 
TEST  AND  EVALUATION  ELEMENTS 
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Utilization  of  Simulation  Techniques 

Early  testing  of  system  characteristics  and  specific  subsystems 
usually  require  utilization  of  same  form  of  simulation  techniques.  Therefore, 
more  effective  utilization  of  models,  test  simulators,  and  test  measurement 
techniques  is  essential  if  testing  complexity  and  costs  are  to  be  kept 
under  control.  The  establishment  of  permanent  testbeds  for  this  type  of 
testing  is  possible  and  certainly  appropriate  if  they  can  be  used  on  a 
recurring  basis.  Even  though  simulators  and  testbeds  may  not,  in  all 
cases,  accurately  project  operational  suitability  characteristics  in 
absolute  terms,  they  are  extremely  useful  in  showing  the  relative  impact 
of  deficiencies,  quantifying  risks  and  identifying  critical  issues.  Close 
coupling  of  realistic  field  testing  with  simulators  and  testbeds  is  an 
essential  technique  for  inproving  test  accuracy.  This  type  of  testing 
provides  the  developer  and  user  with  alternatives  for  support  in  a  dynamic 
and  interactive  manner. 

Definition  of  Simulation  Techniques 

For  purposes  of  simplif ication,  three  distinct  types  of  simulations 
are  identified  for  test  and  evaluation:  (1)  computer  simulations,  (2) 
system  testbeds  and  (3)  system  prototypes.  The  distinguishing  feature 
of  computer  simulations  is  that  they  are  strictly  mathematical  procedural 
representations  of  systems  and  do  not  employ  any  actual  hardware.  They 
may,  however,  incorporate  some  of  the  actual  software  that  might  be  used 
in  a  system.  The  digital  computer  simulations  used  to  generate  performance 
contours  for  air-to-air  missiles  are  examples  of  this  class  of  simulation. 

The  second  generic  type  of  simulation  is  the  system  testbed.  A  systen 
testbed  usually  differs  from  a  computer  simulation  in  that  it  contains 
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some,  but  not  necessarily  all,  of  the  actual  hardware  that  will  be  a 
part  of  the  system.  Other  elements  of  the  system  are  either  not  incorporated 
or  are  incorporated  in  the  form  of  computer  simulations.  The  system 
operating  environment  (including  threat)  may  either  be  physically  simulated, 
as  in  the  case  of  a  flying  testbed,  or  it  may  be  computer  simulated,  as 
in  the  case  of  a  laboratory  testbed.  Aircraft  cockpit  simulators  used 
to  evaluate  pilot  performance  are  good  examples  of  system  testbeds.  The 
third  type  of  simulation  used  in  test  and  evaluation  is  the  system  prototype. 
Unlike  the  system  testbed,  all  subsystems  are  physically  incorporated  in 
a  system  prototype.  The  system  prototype  may  ocme  arbitrarily  close  to 
representing  the  final  system  configuration,  depending  on  the  state  of 
development  of  the  various  subsystems  which  compose  it.  Preproduction 
prototype  missiles  and  aircraft  used  in  operational  testing  by  the  Services 
are  exanples  of  this  class  of  simulation.  It  should  be  recognized  that 
the  distinctions  made  between  the  three  generic  types  of  simulation  can 
often  become  blurred.  A  system  testbed,  for  example ,  which  incorporates 
most,  but  not  all,  of  a  system's  hardware  components  can  come  very  close 
to  representing  a  system  prototype.  On  the  other  hand,  a  system  testbed 
that  incorporates  only  one  or  a  few  hardware  subsystems,  with  the  rest 
being  simulated,  takes  on  many  of  the  characteristics  of  a  computer  simulation. 
There  is,  therefore,  a  continuous  spectrum  of  simulation  types  with  the 
pure  computer  simulation  at  one  end  and  the  pure  hardware  prototype  at 
the  other  end,  as  shown  in  Figure  5. 

Role  of  Each  Simulation  Technique 

Each  of  the  three  generic  types  of  simulation  has  various  advantages 
and  disadvantages  depending  on  the  rating  criterion  being  considered. 


APPLICATION  OF  SIMULATION  TECHNIQUES 

IN  SYSTEM  EVALUATION 


SYSTEM  LIFE  TIME 
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1.  Availability 

Early  in  a  system's  life  cycle,  computer  simulations  can  be  expected 
to  provide  the  most  system  evaluation  information.  As  development  of  a 
system  progresses,  more  and  more  subsystems  become  available  in  hardware 
form  and  eventually  all  subsystems  will  become  available  for  incorporation 
in  one  or  more  system  prototypes.  Operational  testing  of  these  prototypes 
frequently  provides  much  of  the  system  evaluation  information  needed  for 
a  decision  on  full  scale  production  and  deployment.  Subsequently,  more 
representative  initial  production  prototypes  will  beocroe  available  for 
testing  to  verify  correction  of  deficiencies,  investigate  evolutionary 
improvements,  develop  tactics,  etc. 

2.  Realism 

A  second  important  factor  of  interest  is  the  degree  of  realism  offered 
by  various  simulation  techniques.  Realism  is  important  because  the  validity 
of  the  results  obtained  from  a  simulation  is  directly  related  to  how 
realistically  the  simulation  represents  the  system  and  its  environment. 

In  general,  prototype  testing  in  an  environment  physically  representative 
of  combat  provides  the  greatest  amount  of  realism  of  the  three  simulation 
techniques.  Testbeds  and  computer  simulations  provide  respectively  less 
realism.  However,  there  are  exceptions  to  this  rule.  A  computer  model, 
for  example,  can  sometimes  more  realistically  represent  the  enemy  threat 
in  those  cases  where  physical  simulations  of  the  threat  are  not  available 
for  prototype  testing.  Also,  a  computer  simulation  can  frequently  provide 
a  more  realistic  representation  of  casualty  effects  than  can  prototype 
testing.  In  most  cases,  however,  prototype  testing  will  be  the  most 
realistic  simulation  technique  available. 
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3.  Believability 

Closely  related  to  the  realism  factor  is  the  acceptance  or  believability 
of  results  obtained  from  simulations.  Decision  makers  will  usually  feel 
more  confident  about  system  evaluation  data  obtained  fran  hardware  prototype 
testing  than  that  from  a  computer  simulation.  It  is  generally  much  easier 
to  mentally  extrapolate  from  a  realistic  prototype  test  of  expected  combat 
performance  than  it  is  from  a  computer  simulation.  Obtaining  a  high 
level  of  confidence  in  the  results  of  a  computer  simulation  normally 
requires  a  detailed  understanding  of  the  assumptions  and  analytical  procedures 
used  in  constructing  the  simulation.  Acquiring  such  an  understanding 
takes  more  time  and  effort  than  is  available  to  most  decision  makers. 

In  many  cases,  the  believability  of  computer  simulations  can  be  greatly 
enhanced  when  these  simulations  have  been  validated  by  corroboration 
with  results  from  realistic  physical  testing. 

4.  Cost 

The  oost  of  computer  simulation  will  usually  be  much  less  than  that 
of  physical  testing,  particularly  where  destructive  testing  of  prototypes 
is  involved.  A  live  warhead  firing  of  a  prototype  air-to-air  missile 
against  a  realistic  full  scale  target  on  an  instrumented  range  could 
involve  millions  of  dollars  of  direct  and  indirect  costs.  The  same  missile 
launching  carried  out  on  a  computer  simulation  would  be  an  order  of  magnitude 
less  costly. 

5.  Time  of  Execution 

Computer  simulation  holds  the  same  advantage  over  physical  testing 
when  it  comes  to  time  of  execution.  Again,  consider  the  case  of  an  air- 
to-air  missile.  Since  an  air-to-air  missile  operates  in  three  dimensional 
airspace,  there  are  an  extremely  large  number  of  altitude  differentials, 
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speed  differentials,  aspect  angles  and  launch  ranges  that  should  be  con¬ 
sidered  in  assessing  its  effectiveness.  Any  attempt  to  physically  test 
the  missile  over  any  reasonably  representative  sample  of  tactical  situations 
would  be  time  consuming,  not  to  mention  expensive.  Simulating  these 
cases  on  a  high-speed  digital  computer,  however,  could  be  accomplished 
in  a  relatively  short  time. 

6.  Control 

Another  factor  in  evaluating  simulation  techniques  is  the  degree  of 
control  that  can  be  exercised  over  relevant  variables.  Here,  there  are 
two  aspects  to  be  considered.  In  one  case,  all  variables  may  be  rigidly 
controlled  to  isolate  the  effects  of  one  particular  variable.  In  a  digital 
computer  simulation  it  is,  of  course,  very  easy  to  control  all  variables. 

In  prototype  testing,  and  particularly  prototype  operational  testing,  it 
is  much  more  difficult.  This  is  due  partly  to  the  sometimes  unpredictable 
behavior  of  men,  machines  and  the  environment,  and  partly  to  the  inability 
to  measure  adequately  all  variables  that  affect  the  outcome  of  the  test. 

7.  Free-Play 

The  second  aspect  of  variable  control  that  should  be  considered  is 
the  amount  of  free-play  permitted  by  a  particular  simulation.  In  a  computer 
simulation  free-play,  if  any,  is  limited  to  those  elements  designed  into 
it  by  the  programmers.  In  operational  testing  of  prototypes,  however,  a 
great  deal  of  free-play  can  be  introduced  by  allowing  players  to  react 
to  situations  in  a  manner  that  each  perceives  to  be  optimal.  The  benefit 
of  free-play  is  that  it  can  expose  problems  not  predicted  in  designing  a 
system  or  constructing  a  computer  simulation.  As  an  example,  free-play 
testing  can  reveal  unexpectedly  effective  enemy  countermeasures  or  expose 
very  low  reliability  in  systems  that  were  previously  thought  to  be  highly 
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reliable.  Figure  6  shows  a  comparison  of  the  three  generic  types  of 
simulation  for  each  of  the  above  evaluation  factors.  Computer  simulations 
and  system  prototypes  will  usually  be  either  best  or  worst  for  each  factor 
considered  with  system  testbeds  somewhere  in  between.  What  is  not  obvious 
from  comparing  the  advantages  and  disadvantages  of  each  individual  simulation 
technique  is  the  synergistic  effect  that  will  usually  result  from  employing 
all  three  of  these  techniques.  For  example,  computer  simulations  can  be 
used  to  pinpoint  critical  performance  zones  that  should  be  explored  by 
physical  testing  of  testbeds  and  prototypes.  This  can  greatly  increase 
the  economy  and  efficiency  of  physical  testing.  On  the  other  hand,  results 
of  physical  testing  can  be  used  to  validate  computer  simulations,  thereby 
increasing  their  accuracy  and  believability.  Once  validated,  computer 
simulations  can  be  used  to  extrapolate  the  results  of  physical  testing 
to  cover  a  great  many  more  scenarios  and  engagement  situations,  than 
would  be  possible  from  physical  testing  alone.  The  end  result  of  these 
synergistic  effects  is  that  the  value  of  using  a  combination  of  the  three 
types  of  simulation  techniques  can  greatly  exceed  the  value  of  using  any 
single  one. 

Current  System  Acquisition  Trends 

There  are  several  trends  in  system  acquisition  policy  currently 
influencing  DoD  decision  techniques. 

1.  Shorter  Acquisition  Cycle 

One  of  the  more  significant  trends  has  been  an  increased  emphasis 
on  shortening  the  acquisition  cycle.  Shortening  the  acquisition  cycle 
has  two  important  implications.  First,  it  reduces  the  length  of  time 
available  for  obtaining  system  evaluation  data.  Second,  it  requires 
that  quality  system  evaluation  data  be  made  available  earlier  in  the 
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acquisition  cycle.  Requiring  system  evaluation  data  earlier  in  the  acquisition 
cycle  likewise  implies  a  greater  emphasis  on  computer  simulations  and 
system  testbeds  versus  system  prototypes.  This  is  the  case  because  computer 
simulations  and  system  testbeds  will  normally  become  available  much  earlier 
in  a  system' s  development  cycle  than  will  system  prototypes.  Computer 
simulations  can  be  developed  f rcm  basic  preliminary  design  data  and 
extrapolations  from  computer  simulations  of  similar  predecessor  systems. 
Representative  prototypes,  on  the  other  hand,  will  not  normally  become 
available  until  system  hardware  development  is  virtually  complete. 

Therefore,  conducting  prototype  testing  will  usually  involve  extensive 
planning,  preparation,  training  and  set-up  time.  In  addition,  if  only 
one  or  a  few  prototypes  are  available,  test  replications  have  to  be 
conducted  in  series.  By  contrast,  multiple  test  replications  on  a  computer 
simulation  can  be  achieved  in  much  less  time  and  w*th  a  minimum  of  lead 
time. 

2.  Increasing  System  Cost 

The  increasing  cost  of  weapon  systems  is  another  trend  affecting 
the  use  of  simulation  to  obtain  system  evaluation  data.  As  the  capabilities 
of  our  ships,  aircraft,  missiles  and  other  weapon  systems  have  increased 
over  the  years,  so  have  their  development  and  procurement  costs.  The 
result  has  been  to  increase  greatly  the  cost  of  prototype  testing, 
particularly  destructive  prototype  testing.  The  cost  of  computer  simulation 
has  also  risen  over  the  years,  but  not  nearly  as  much  because  of  the 
technological  and  programing  efficiencies  introduced  in  the  computer 
field.  The  net  result  is  that,  as  cost  containment  priority  increases, 
more  emphasis  is  placed  on  computer  simulation  to  quantify  risks. 
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Increasing  Complexity 
Increasing  complexity  is  another  trend  that  mast  be  considered. 

The  increasing  capabilities  demand  of  our  weapon  systems  have  required 
ever-increasing  sophistication  and  complexity  in  their  design.  The  situation 
is  compounded  for  the  test  and  evaluation  ccrmunities  by  complex  threats 
and  the  difficult  environments  in  which  they  must  operate.  Increasing 
complexity  makes  it  much  more  difficult  to  predict  how  a  system  will 
perform  against  the  threat  in  an  operational  environment.  This  argues 
for  realistic  field  testing  of  prototypes,  which  is  frequently  the  only 
way  to  ensure  proper  consideration  of  the  multitude  of  factors  that  impact 
the  performance  of  a  modern  weapon  sys tern . 

Research  and  Development  Initiatives 

Looking  toward  future  use  of  simulation  techniques,  DOT&E  has  initiated 
research  to  determine  the  feasibility  of  networking  geographically  dispersed 
systems.  If  these  new  initia:ives  are  successful,  several  important  T&E 
benefits  may  result,  including: 

o  The  ability  to  simulate  increasingly  complex  systems  and  scenarios, 
o  Reduced  time  and  cost  associated  with  the  acquisition  of  new 
systems  through  multiple  and  repetative  use  of  test  facilities. 

o  Increased  confidence  on  the  part  of  decision  makers  due  to  increased 
realism  in  the  simulation. 

Interdependent  simulations  must  be  able  to  exchange  information  in 
a  timely  manner  and  the  amount  of  information  that  must  be  exchanged  may 
be  quite  substantial.  The  emerging  technology  of  wideband  communications 
makes  the  transmission  of  large  volumes  of  information  more  attractive. 
However,  this  technology  does  not  eliminate  many  of  the  historical  problems 
such  as  communication  delays  and  information  security  associated  with 
both  satellite  and  terrestrial  microwave  links. 
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In  recognizing  these  potentials  and  associated  problems,  we  are 
attempting  to  develop  a  set  of  guidelines  for  interconnected,  geographically 
distributed  simulations  of  the  previously  described  generic  types.  This 
effort  can  serve  as  the  basis  for  a  future  demonstration  effort.  To 
achieve  this  goal,  we  have  established  the  following  specific  objectives: 

o  Identification  of  classes  of  simulations  for  which  distribution 
of  parts  of  the  simulation  would  be  advantageous 

o  Delineation  of  the  implication  of  new  computer  and  communications 
technologies  as  related  to  geographically  distributed  simulations 

o  Development  of  a  preliminary  design  description  for  networking 
simulations 

o  Application  of  the  design  guidelines  including  standards,  to 
determine  the  feasibility  of  interconnecting  specific  simulations. 

o  The  implementation  of  a  database  management  system  to  catalog 
information  on  available  models,  data,  hardware  specifications,  and  testbed 
architectures. 

Different  systems  within  the  Department  of  Defense  will  be  analyzed 
as  possible  candidates  for  inclusion  into  geographically  distributed 
simulation  networks.  These  simulations  will  be  categorized  along  several 
dimensions  and  technological  advances  expected  to  influence  successful 
construction  of  geographically  distributed  simulation  networks.  Alternatives 
will  be  examined  to  assess  the  impact  of  advances  in  hardware,  software 
and  communication  technologies. 

Some  joint  test  programs  are  also  attempting  to  connect  geographically 
dispersed  simulation  facilities.  For  example,  the  Identification  Friend, 

Foe  or  Neutral  (IFFN)  joint  test  has  been  established  to  evaluate  the 
capability  of  the  current  NATO  Air  Defense  System  and  proposed  modifications 


to  assist  in  the  identification  of  airborne  targets.  The  IFFN  joint 
test  will  use  a  distributed  network  in  which  various  simulation,  test 
and  operational  facilities  across  the  country  are  linked  together  on  a 
realtime  basis  through  a  central  computer  facility. 

The  Department  of  Defense  appears  to  be  moving  toward  greater  use 
of  simulation  techniques  in  its  decision  making.  This  trend  is  becoming 
especially  pronounced  in  the  Ccrmand,  Control  and  Communications  (C3I) 
area,  since  simulation  techniques  can  be  readily  applied  to  the  tele¬ 
processing  technology  and  doctrine  employed  by  C3  systems.  However, 
limited  advances  have  been  made  in  "use  of  distributed  simulation”. 

Essential  to  future  success  is  closer  coupling  of  simulation  facilities 
and  functions.  An  architecture  with  selectable  centralized  control  and 
distributed  operation  is  certainly  a  feasible,  worthwhile  objective.  A 
distributed  C3  test  bed  using  already-developed  simulations  and  facilities 
in  place  could  well  became  a  national  asset  that  could  be  used  for  test 
and  evaluation  of  individual  service  or  joint  programs. 

TECHNOLOGY  IMPLICATIONS 

Closely  linked  to  all  the  previously  described  possible  improvements 
in  T&E  are  the  technologies  available  for  transfer,  storage  and  processing 
of  information.  Although  such  technologies  span  a  large  spectrum,  only 
those  expected  to  be  major  drivers  within  the  next  few  years  will  be 
discussed  in  this  report. 

Trends  In  Information  Transfer 

The  information  society  is  rapidly  becoming  a  reality  as  microelectronic 
computers  and  other  electronic  disciplines  provide  unprecedented  universal 
wideband  networks.  Along  with  this  evolution,  the  meshing  between  computer 


and  communication  equipment  has  become  so  pervasive  that  communication 
specialists  have  coined  new  words  such  as  " ccmpunicat ions , "  " telematique, 
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"telematics"  and  "teleprocessing." 

The  term  "teleprocessing"  utilized  in  the  context  of  this  paper  is 
defined  as  the  combined  use  of  communications  facilities  and  data  processing 
equipment.  Dominant  among  the  many  elements  supporting  this  definition 
cure  (a)  technological  advances  that  have  tended  to  blur  traditional  lines 
of  demarcation  between  communications  and  processors  and  (b)  recent  judicial 
decisions  rendered  in  cases  such  as  IBM  and  AT&T.  Technology  during  the 
next  several  years  will  continue  to  support  such  transparency  as  tele¬ 
processing  services  become  more  widespread.  The  major  inhibitor  to  such 
growth  is  the  scientific  limitations  in  software  development. 

Major  Technologies 

The  technologies  that  have  already  induced  major  changes  in  tele¬ 
processing  systems  are  very-large-scale  integration  (VLSI)  circuits, 
digital  techniques,  satellites,  and  optical  fibers.  Transition  of  these 
technologies  has  and  will  continue  to  take  many  forms  as  various  elements 
are  designed  to  provide  reliable  and  timely  teleprocessing  services  in  a 
cost  effective  manner  over  networks  consisting  of  both  space  and  terrestrial 
systems .  The  driving  force  behind  this  change  is  the  enormous  demand 
for  economical  information  transfer  services.  The  impact  of  the  micro¬ 
electronics  revolution  on  these  services  is  just  beginning  to  be  felt 
with  computing  hardware  costs  dropping  sharply.  The  widespread  utilization 
of  the  microprocessor  is  due  to  the  fact  that  it  is  inexpensive  hardware 
and  can  be  programmed  to  perform  an  unlimited  variety  of  tasks.  This 
permits  more  and  more  software  to  be  written  for  more  and  more  embedded 
computer  applications.  However,  this  has  created  a  significant  delima 
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in  that  software  continues  to  be  more  of  an  art  than  science  and  is 
considered  to  be  a  major  contributor  to  increased  risk  in  almost  all 
systems.  Such  is  the  case  in  both  caimercial  and  defense  systems  with 
embedded  computers  already  becoming  a  dominant  element  of  the  total  system 
investment  and  software  a  significant  factor  in  performance  deficiencies 
and  cost  increases. 

Integrated  circuits  have  already  induced  major  changes  in  weapon 
system  developments  and  testing  techniques  and  will  continue  to  do  so 
during  the  next  decade.  Wafer  size  is  increasing,  allowing  for  greater 
increases  in  the  number  of  devices  that  can  be  produced  on  a  single  chip. 

As  chips  get  larger  and  density  increases,  a  new  generation  of  integrated 
circuits  will  be  possible  wiyi  one  million  devices  on  a  single  chip  now 
considerd  to  be  within  reach  and  one  billion  components  per  chip  being 
discussed  as  possible  in  the  future.  The  amount  of  memory  on  a  single 
chip  has  been  quadrupling  every  two  years  and  is  expected  to  continue 
with  chips  containing  8  to  32  megabites  of  memory  within  the  next  decade. 
Sixty- four  kilobit  random  access  memory  chips  are  new  being  shipped  in 
volume.  The  small  size,  high  reliability,  and  low  power  consumption  of 
these  components  make  them  suitable  for  numerous  test  and  evaluation 
applications.  New  logic  inside  future  chips  will  increase  the  speed  of 
microprocessors  and  the  large  number  of  functions  on  a  single  chip  will 
also  greatly  increase  microprocessor  design  options.  Programmed  with 
high  level  languages,  systems  will  feature  minicomputer-like  architectures 
and  powerful  instruction  sets  which  will  allow  written  software  to  be 
executed  in  any  location  of  the  microcomputer's  address  map. 

Implicit  in  these  developments  is  the  continually  increasing  importance 
of  software  that  is  embedded  in  the  computer's  architecture.  Essential 
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to  improvements  in  software  testing  is  the  early  involvement  of  the  testing 
community  in  the  development  cycle.  Such  involvement  permits  all  software 
life  cycle  activities  to  be  evaluated  in  a  timely  manner  parallel  to 
hardware  and  assists  in  ensuring  that  software  specifications  and  testing 
include  performance,  reliability  and  implementation  considerations.  This 
integrated  approach  is  expected  to  have  high  payoff  in  the  future  with 
significant  reductions  in  endpoint  testing  problems.  The  introduction 
of  microprocessors  has  already  permitted  major  advancements  in  instru¬ 
mentation  and  test  methodology.  For  example,  technology  now  permits 
test  data  to  be  collected  on  a  distributed  realtime  basis  and  automatically 
relayed  to  a  central  computer  that  has  built-in  software  routines  for  a 
rapid  analysis  and  display  of  results.  Test  and  diagnostics  on  many 
small  systems  are  now  being  handled  automatically  on  a  routine  basis. 
However,  large  software  systems  continue  to  present  problems. 

Progress  in  lew-loss  optical-fiber  technology  and  integrated  circuits 
is  providing  a  new  generation  of  digital  transmission  systems.  Such 
progress  is  a  vital  part  of  the  move  toward  time-shared  digital  networks. 
Advances  in  both  terrestrial  and  satellite  systems  are  also  making  important 
contributions  to  the  development  of  an  integrated  network  with  transmission 
speeds  varying  from  a  few  kilobits  per  second  to  hundreds  of  megabits 
per  second.  As  a  result  of  these  improvements,  networks  have  the  potential 
of  becoming  universal  translators  of  signals  with  transparent  interfaces. 
System  Relationships 

The  interaction  of  telecommunications  and  computing  into  a  new 
teleprocessing  form  of  service  has  placed  new  demands  on  the  proess  of 
achieving  point-to-point  communications  effectively  and  efficiently. 
Connection  of  direct  electrical  channels  between  terminals,  typical  of 


telegraphic  or  telephonic  services,  has  been  augmented  by  methods  of 
assigning  time  intervals,  or  time  slots,  on  a  single  channel  to  subscribers 
on  demand.  The  time  assignment  method  of  networking  is  a  feature  of 
emerging  sa tel ite- based  systems  and  packet  switching  systems. 

Networks  must  be  able  to  interact  with  a  multiplicity  of  voice 
digitization  techniques,  digital  data  techniques  and  analog  techniques. 
Transmission  rates  must  be  accomodated  ranging  from  kilbits  per  second 
to  megabits  per  second.  Services,  such  as  low  speed  video  and  facsimile, 
currently  practical  almost  solely  via  dedicated  circuits,  must  be  integrated 
on  a  system  basis.  Data  communication  system  must  be  able  to  handle 
interactive,  query/response,  data  base  update,  and  bulk  data  traffic 
interchangeably  and  efficiently.  The  systems  must  provide  adaptability 
for  the  efficient  use  of  trunk  capacity  for  a  wide  variety  of  transmission 
characteristics  covering  a  wide  range  of  transmission  rates.  They  must 
be  compatible  not  only  with  established  communications  links  (wireline, 
line-of-sight  radio,  satellite  radio)  but  also  new  communications  links 
based  on  techniques  such  as  time  division  multiple  access  satellite  and 
various  types  of  microwave  and  optical  links.  They  must  be  capable  of 
maintaining  link  and  large  scale  network  sychronization  with  practical 
networks  incorporating  such  factors  as  expandability,  modularity,  avail¬ 
ability,  and  interoperability  as  well  as  being  economical  in  transmission 
costs  by  utilizing  transmission  capcity  efficiently.  A  wide  variety  of 
differing  terminal  types  will  be  utilized  with  transmission  rates  ranging 
from  hundreds  of  bits  per  second  to  hundreds  of  kilobits  and  transaction 
sizes  ranging  from  several  hundred  bits  to  millions  of  bits  of  data. 

Wideband  networks  have  the  potential  for  providing  a  new  dimension 
to  test  facility  utilization  with  direct  connectivity  permitting  more 
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synergistic  relationships.  The  TRI-TAC  Communications  Test  Facility  at 
Fort  Huachuca,  Arizona,  and  the  Ccrmand,  Control  and  Communications  Systems 
Integration  Test  and  Evaluation  Site  at  the  Naval  Ocean  Sy starts  Center, 

San  Diego,  California,  are  examples  of  test  facilities  attempting . to 
utilize  such  networks.  These  facilities  assist  in  filling  the  gap  between 
the  developer  and  the  user  by  providing  interoperability  on  a  multi¬ 
system,  multi-platform  basis.  As  previously  indicated,  Joint  Test  programs 
are  also  being  designed  to  carry  this  concept  even  further  with  a  test 
bed  now  being  designed  to  conform  to  a  centrally  controlled  distributed 
architecture.  A  computer  network  can  also  provide  a  basic  framework  for 
developing  and  oonnecing  geographically  dispersed  simulation  facilities. 

The  network  should  be  thought  of  as  a  delivery  service  that  guarantees 
the  timely  delivery  of  information  between  sites  served  by  a  shared  switched 
system  vice  dedicated  circuits.  Prior  to  effective  Implementation  of 
such  a  system,  numerous  issues  such  as  simulation  component  differences, 
interfaces  and  communications  medium  constraints  must  tie  resolved. 

CONCLUSIONS 

Inadequate  planning  and  information  flow  are  identified  in  this 
report  as  major  contributors  to  T&E  associated  acquisition  problems.  In 
order  to  assist  in  correcting  this  situation,  a  concept  is  developed  for 
an  integrated  T&E  Decision  Support  System.  This  concept  envisions  an 
effective  and  timely  processing  of  data  accommodating  the  cognative  styles 
and  judgemental  heuristics  of  decision  makers.  It  is  the  contention  of 
this  author  that  such  influences  may  be  the  most  important  factor  in  the 
decision  process  at  the  higher  levels  in  DoD. 
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A  draft  proposal  of  a  generic  model  is  provided  as  an  appendix  to 
this  report.  These  T&E  processes  and  associative  acquisition  relationships 
are  considerd  to  provide  essential  building  blocks  required  for  a  DSS. 

Such  effort  must  also  include  more  effective  utilization  of  test  resources, 
i.e.,  computer  simulations.  Much  of  the  differentation  that  has  historically 
inhibited  T&E  effectiveness  could  possibly  be  neutralized  with  the  concepts 
and  techniques  suggested.  These  concepts  are  considered  to  be  in  full 
support  of  acquisition  trends  in  DoD  and  within  state-of-the-art  technology. 
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GENERIC  MODEL  OF  DEFENSE  TEST  AND  EVALUATION 


A  model  that  specifies  the  policies,  processes  and  summary  data  requirements 
from  independent  test  activities  and  development  ccmunities  would  assist 
in  improving  the  information  flow  and  be  a  major  contributor  to  increasing 
the  effectiveness  of  the  acquisition  process  for  major  weapon  systems. 

Such  an  approach  should  assist  in  reducing  the  time  to  complete  the  acquisition 
process.  However,  closely  coupled  with  this  model  must  be  the  establishment 
of  distributed  data  bases  that  can  be  selectively  utilized  by  the  development 
and  testing  canmunities.  Equally  important  is  incorporation  of  these 
factors  in  plans  developed  during  early  phases  of  the  acquisition  cycle. 

Such  planning,  when  placed  in  perspective  with  a  balanced  testing  program, 
can  be  an  effective  tool  to  assist  in  solving  the  critical  issues  that 
are  precluding  vital  improvements  in  the  performance  and  operational 
readiness  of  our  deployed  forces.  More  effective  advanced  planning  and 
budgeting  would  also  permit  a  more  timely  and  synergistic  utilization  of 
simulators,  test  beds  and  prototype  equipments  throughout  the  acquisition 
cycle.  Such  an  approach  is  essential  for  large  multiple-mission  platforms 
that  must  be  developed  over  an  extended  period  of  time. 
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DIRECTOR  DEFENSE  TEST  AND  EVALUATION 


MISSION:  Manage  Department  of  Detense  Test  and  Evaluation  (T&E) 

Activities 

OBJECTIVE:  Primary  objective  of  Defense  T&E  is  to  support  the  acquisition 
of  reliable  and  effective  weapon  systems  for  the  operating  forces.  Director 
Defense  T&E  will  manage  test  activities  to  accomplish  the  following: 

o  Provide  an  assessment  of  weapon  system  operational  effectiveness 
and  suitability  at  major  milestones  to  demonstrate  that  the  technical 
performance  specifications  have  been  met  and  to  show  that  engineering 
design  is  satisfactory. 

o  Support  Service  Independent  Test  Agency  efforts  to  maintain  strong 
checks  and  balances  in  the  acquisition  process  by  testing  and  evaluating 
weapon  systems  in  their  expected  operational  environment. 

FUNCTIONS: 

o  Formulate  and  implement  T&E  policy. 

o  Develop  and  submit  to  SECDEF,  Acquisition  Executive  and  DSARC 
Principals  T&E  assessments  at  major  program  milestones. 

o  Direct  utilization  of  resources  and  management  of  major  DoD  Facilities, 
o  Periodically  review  and  assess  developmental  test  and  evaluation 
(OT&E)  and  operational  test  and  evaluation  (DT&E)  for  major  programs 
(non-DSARC  level) . 

o  Evaluate  arvd  approve  Test  and  Evaluation  Master  Plans  (TEMPs) . 
o  Sponsor  and  fund  Joint  Operational  Test  and  Evaluation  (JOT&E) . 
o  Conduct  Mission  Area  evaluations. 

o  Manage  and  fund  the  Foreign  Weapon  Evaluation  Program, 
o  Coordinate  International  T&E  Memorandums  of  Understanding  (MDUs) 


and  agreements. 
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POLICY  FORMULATION  AND  IMPLEMENTATION  (4.1) 


POLICY 

o  Each  DcD  official  who  has  direct  or  indirect  responsibility  for 
the  acquisition  process  shall  be  guided  by  the  policies  and  objectives 
of  QMB  Circular  A-109  for  major  system  acquisitions. 

o  The  SECDEF  will  be  advised  by  the  Defense  Acquisition  Executive 
(DAE)  in  making  all  of  the  major  decisions. 

o  DDT&E  will  monitor  testing  progress,  develop  an  independent  assessment, 
and  report  findings  to  DAE  and  SECDEF. 

o  T&E  shall  begin  as  early  as  possible  and  be  conducted  throughout 
the  acquisition  process  to  assess  and  reduce  risks  and  estimate  operational 
effectiveness  and  suitability. 

o  Meaningful  critical  issues,  test  objectives,  and  evaluation  criteria 
related  to  the  satisfaction  of  mission  need  shall  be  established  before 
tests  begin. 

o  Successful  planning  for  and  accomplishment  of  T&E  objectives 
will  be  a  key  requirement  for  decisions  to  commit  significant  additional 
resources  to  a  program  or  to  advance  it  from  one  acquisition  phase  to 
another. 

o  Dependence  on  subjective  judgement  concerning  system  performance 
shall  be  minimized  —  to  the  extent  possible  appropriate  test  instrumenta¬ 
tion  will  be  used  to  provide  quantitative  data  for  system  evaluation. 

PROCESS  DESCRIPTION 

A  Service,  combining  operational  need,  technical  feasibility  studies 
and  cost  estimates  develops  the  Justification  for  Major  System  New  Start 
(JMSNS) .  This  document,  coordinated  within  and  approved  by  OSD,  includes 
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background  information  and  justification  for  the  proposed  system.  Also, 
par  of  the  PCM  submittal  is  intended  to  provide  for  meaningful  reviews 
of  alternatives  and  to  provide  for  adequate  front-end  funding  of  test 
articles.  DDT&E,  other  than  coordination  on  the  JMSNS  and  subsequent 
program  documents,  does  not  become  directly  involved  in  further  control 
until  Service  submission  of  the  TEMP,  which  is  part  of  the  preparation 
for  DSAHC  I. 

DoDD  5000.3  covers  the  procedures  for  development  and  submission  of 
TEMPs  and  is  the  major  policy  formulation  document  for  DDT&E.  This  policy 
is  implemented  through  the  major  acquisition  reviews  and  procedures  described 
in  DoD  Directives  5000.1  and  5000.2.  In  addition,  DDT&E  enforces  and 
formulates  specific  requirements  through  assessment  reports  to  the  Secretary 
of  Defense,  which  are  part  of  the  SDDM  submission. 

Annually,  DDT&E  conducts  a  Test  Commanders  Conference  where  areas 
requiring  resolution,  added  emphasis,  or  coordination  are  discussed.  In 
addition,  DDT&E  memoranda  may  be  used  for  specific  requirements  or  policy 
of  limited  duration. 


POLICY  FORMULATION  AND  IMPLEMENTATION 
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MISSION  AREA  ANALYSIS  (4.2) 


POLICY 

o  All  OSD  offices  submit  inputs  (problem  areas,  issues,  deficiencies 
and  alternative  concepts)  to  respective  Mission  Area  Coordinators  (MACS) . 

o  MACS  coordinate  inputs,  develop  ini tied,  assessments  and  submit 
to  USDRE  (Plans  and  Development) . 

o  USDRE (P&D)  collects  inputs  and  cost  alternatives,  and  develops 
Mission  Area  Assessment  (MAA) . 

o  USDC2  and  MRA&L  present  MAA  to  the  Defense  Resources  Board  (DRB) . 
o  USDRE (P&D)  incorporates  approved  MAA  into  formulation  of  the 
Draft  Defense  Guidance. 

o  DRB  reviews  Draft  Defense  Guidance, 
o  SECDEF  issues  Defense  Guidance. 

PROCESS  DESCRIPTION 

Mission  Area  Analysis  is  really  a  continuing  process  with  the  formal 
analysis  done  each  Fall  as  OSD  develops  the  Defense  Guidance.  Certainly 
the  same  data  is  utilized  in  deciding  whether  to  increase  or  decrease  a 
program's  funding  level  or  whether  to  ask  Congrss  for  supplemental  funds. 
Nevertheless,  in  the  Fall  this  process  is  formalized  as  each  office  within 
OSD  is  asked,  considering  the  current  strategy,  to  offer  problem  areas, 
issues,  deficiencies  and  alternative  concepts  whereby  DoD  may  achieve 
itsstated  strategy  objectives. 

DDT&E's  input  is  of  primary  interest  in  that  this  input  should  highlight 
program  deficiencies,  operational  effectiveness,  and  operational  suitability. 
For  example,  programs  must  provide  adequate  front-end  funding  for  early- 
on  testing,  i.e.,  adequate  test  runs  and  test  hardware.  It  may  be  necessary 


to  consider  foreign  weapons  as  alternatives.  DDT&E  should  encourage 
evolutionary  developments  as  a  nv»'ns  of  decreasing  weapon  development 
time  and  cost.  The  resultant  analyses  are  forwarded  to  a  Mission  Area 
Coordinator  (MAC)  whose  task  is  to  combine  inputs  and  develop  a  program 
which  most  cost  effectively  supports  the  strategy. 

Within  USDRE,  each  MAC  then  forwards  the  MAA  to  USDKE  (P&D)  where 
the  program  is  combined  and  costed  to  ensure  affordability.  USDRE (P&D) 
coordinates  the  MAA  with  MRA&L  and  presents  draft  MAA  to  the  DKB.  The 
ISC  can  approve  the  MAA  and  forward  for  inclusion  in  developing  the 
Defense  Guidance  or  it  can  provide  additional  guidance  and  direct  a 


resubmission. 


mission  area  analysis  third-level 
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TEST  AND  EVALUATION  ASSESSMENT  (4.3) 


POLICY 

o  DDT&E  will  monitor  testing  progress  on  a  continuous  basis  throughout 
the  acquisition  process  and  report  his  independent  assessment.  Milestone 
I  reviews  will  evaluate  system  concepts  and  alternatives  based  on  specific 
goals  and  thresholds  established  in  an  approved  TEMP.  Milestone  II  reviews 
will  include  an  assessment  of  previous  by  established  test  plans  and 
test  results.  Milestone  III  reviews  shall  verify  operational  effectiveness 
and  suitability  of  major  weapon  systems. 

o  T&E  ?csessments  shall  include  the  following: 
o  Introduction  and  background  on  program, 
o  Program  documentation, 
o  Testing  accomplished  to  date, 
o  Future  testing. 

o  Analysis  and  discussion  of  issues, 
o  Conclusions  and  recommendations . 

o  Assessment  reporting  shall  include  the  following: 
o  S urinary  report  to  DDT&E. 
o  Independent  report  to  DSARC  principles, 
o  S unmar y  paragraph  in  the  SDCM. 
o  Independent  summary  to  DAE  and  SBCDEF. 
o  Sunnary  on  selected  acquisitions  to  U.S.  Congress, 
o  Various  periodic  summary  status  reports. 

PROCESS  DESCRIPTION 

Primary  inputs  utilized  in  assessments  are: 

o  Test  and  Evaluation  Master  Plan  (TEMP)  and  more  detailed  support 
documents  developed  by  responsible  Service  activities. 


o  Service  Independent  Test  Agency  reports  and  briefings, 
o  Developmental  Test  and  Evaluation  data  from  Service  Program  Managers, 
laboratories,  industry  developers,  etc. 

Assessment  reports  are  provided  to  key  decision  makers  in  OSD  prior 
to  major  program  reviews  or  DSARCs  and  to  the  Deputy  Secretary  of 
Defense/Secretary  of  Defense  for  DSARC  level  decisions.  Critical  to 
successful  assessments  are  reduction  of  volumes  of  data  distributed  over 
long  periods  of  time  to  a  6-12  page  report  that  can  be  further  condensed 
to  a  1  to  2  paragraph  insertion  into  an  SDCM. 


INTERNATIONAL  AND  FOREIGN  WEAPON  EVALUATION  (4.4) 

POLICY 

o  Equipment  procured  for  U.S.  Forces  employed  in  Europe  under  the 
terms  of  the  North  Atlantic  Treaty  should  be  standardized  or  at  least 
interoperable  with  equipment  of  other  members  of  NATO. 

o  Priorities  for  the  DoD  are  established  annually  in  the  consolidated 
guidance.  In  addition,  five  top  priority  areas  for  interoperability  and 
standardization  have  been  established  by  the  JCS  and  endorsed  by  the 
NATO  Military  Committee. 

o  Interoperability  and  standardization  must  be  considered  in  meeting 
worldwide  requirements. 

o  Three  major  U.S.  approaches: 

o  Establishment  of  general  and  reciprocal  procurement  Memoranda 
of  Understanding  "KXJ)  with  NATO  member  nations. 

o  Negotiation  of  dual  production  of  developed  or  nearly  developed 

systems. 

o  Creation  of  families  of  weapons  (program  packages)  for  systems 
not  yet  developed. 

o  Fundamental  to  the  success  of  the  three  major  U.S.  approaches  is 
the  improvement  of  the  management  structure  for  arms  cooperation  within 
the  Alliance. 

o  DoD  shall  not  normally  enter  government- to-government  offset 
procurement  agreements  with  other  nations. 

Appropriate  arrangements  are  to  be  made  to  exchange  cost  data  between 
prospective  government  participants. 

o  Ccnmercial  implications  of  technology  transfers  proposed  in  support 
of  a  collaborative  project  should  be  considered  when  weighing  the  costs 
and  benefits  of  that  project. 


o  In  general,  the  U.S.  shall  permit  sales  and  transfers  by  NATO 


allies - . 

o  DcD  components  shall  encourage  the  transfer  of  technology  - . 

o  General  and  reciprocal  procurement  MOO's  can  provide  for  broad 
access  of  signatory  nations  to  each  other's  acquisition  processes. 

o  The  DoD  may  reduce  or  waive  various  FMS  charges  - . 

o  Arrangements  for  NATO  industrial  participation  are  encouraged, 
o  NATO  standardization  and  interoperability  are  basic  goals  to  be 
included  in  acquisition  programs. 

PROCESS  DESCKITTION 

A  three  phased  evaluation  procedure  is  normally  followed  for  Foreign 
Weapon  Evaluation  (EVE) .  Phase  I  consists  of  the  identification  of  foreign 
systems  which  may  satisfy  a  Service  requirement.  Phase  I  evaluation  is 
normally  initiated  by  a  user  requirement  document  or  a  NATO  standardization 
agreement  (STANAG) .  Phase  I  EVE  may  also  be  initiated  through  liaison 
officers  and  foreign  unsolicited  proposals.  In  the  absence  of  a  Service 
requirement,  the  effort  will  be  terminated.  If  there  is  a  requirement 
but  no  approved  requirement  document,  a  requirement  document  will  have 
to  be  prepared  and  a  full  capability  search  conducted  to  identify  other 
potential  candidates.  The  list  of  potential  candidates  may  include  both 
foreign  and  U.S.  systems .  This  list,  based  upon  available  information, 
should  contain  descriptions  of  the  potential  candiates,  provide  comparisons 
of  performance  parameters  with  the  requirements,  and  provide  unit  cost 
and  status  of  the  systems.  The  list  is  coordinated  with  the  appropriate 
interested  agencies  (Service  headquarters  for  final  approval  in  the  proceedings 
of  an  Army  In-Process  Review,  Ait  Force  Program  Management  Directive 
(PMD)  or  Navy  CNO  Evaluation  Board  (CEB)  decision. 
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X£  the  decision  at  Service  headquarters  is  to  approve  Phase  I  and 
continue  into  Phase  II,  the  development  command  becomes  the  coordinator 
for  the  evaluation  effort.  An  evaluation  coordination  working  group 
(ECWG)  is  established  and  chaired  by  the  development  coordinator,  consisting 
of  the  user's  representative,  the  logistician,  developer,  trainer,  operational 
evaluator,  and  development  evaluator.  As  part  of  Phase  II,  foreign  governments 
and/or  industries  are  contacted  to  apprise  them  of  U.S.  interest  and  to 
ascertain  if  the  foreign  governments  and  industries  are  willing  to  cooperate 
in  the  program.  The  test  issues  are  prepared  by  the  appropriate  evaluating 
agencies  and  submitted  to  the  ECWG.  The  test  issues  are  used  by  developmental 
and  operational  testers  to  identify  the  data  requirements  and  prepare  an 

evaluation  plan.  The  data  search  results  are  submitted  to  the  ECWG  for 

* 

a  decision  as  to  whether  the  data  are  sufficient  for  preparation  of  a 
final  report  by  the  evaluators.  If  data  gap®  exist,  test  items  are  obtained 
from  the  foreign  governments  by  way  of  loan,  lease  or  purchase;  test 
plans  prepared,  and  limited  developmental  and/or  operational  tests  conducted. 

If  testing  is  expected  to  be  outside  the  original  design  envelope  (for 
instance,  in  extreme  cold  and  hot  weather  conditions) ,  it  is  advisable 
to  purchase  the  test  items  so  special  requests  to  the  foreign  developer, 
with  his  veto  authority,  are  not  required,  and  repair/refurbishing  is 
not  required  to  the  test  items.  Developmental  and  operational  testing 
will  normally  be  combined  or  integrated  for  Phase  II.  At  the  end  of  the 
testing  and  evaluation  phase,  test  reports  are  prepared  and  Service  approval 
sought,  for  one  of  the  following  options; 

(1)  To  accept  the  item  as  meeting  U.S.  requirements  and  to  acquire 
the  item  by  direct  buy  or  co-production. 

(2)  To  reject  all  items  and  terminate  the  effort. 


(3)  To  continue  to  monitor  development  of  the  item  or  items  by  the 
foreign  country. 

(4)  To  acquire  additional  data  by  proceeding  with  an  additional 
phase  for  one  or  more  foreign  items. 


I 


JOINT  TEST  AND  EVALUATION  (4.5) 

POLICY 

o  Joint  Test  and  Evaluation  (JT&E)  may  be  initiated  by  DDT&E  to 
accomplish  the  following: 

o  Examine  capability  of  developmental  and  deployed  systems  to 
perform  intended  missions. 

o  Provide  information  for  technical  concepts  evaluation,  system 
requirements,  system  improvements,  system  interoperability,  force  structure 
planning,  developing  or  improving  testing  methodologies  and  obtaining 
information  pertinent  to  doctrine,  tactics,  and  operational  procedures 
for  joint  operations. 

4 

o  Responsibility  for  managing  the  practical  aspects  of  each  JT&E 
will  be  delegated  to  a  specific  DoD  component  and  supported  by  forces 
and  material  from  participating  components. 

o  Control  and  OSD  sponsorship  of  JT&E  will  be  exercised  by  the 
DDT&E. 

o  The  Military  Services,  Commanders  in  Chief  (CINC)  of  the  Unified 
and  Specified  Commands,  and  Joint  Chiefs  of  Staff  may  nominate,  participate 
and  monitor  JT&E  as  appropriate. 

PROCESS  DESCRIPTION 

When  required  and  as  initiated  by  the  DDT&E,  JT&E  will  be  conducted. 

In  addition  to  examining  the  capability  of  developmental  and  deployed 
systems  to  perform  their  intended  mission,  JT&Es  may  also  be  conducted 
to  provide  information  for  technical  concepts  evaluation,  system  requirements, 
system  improvements,  systems  interoperability,  force  structure  planning, 
developing  or  improving  testing  methodologies,  and  obtaining  information 
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pertinent  to  doctrine,  tactics,  and  operational  procedures  for  joint 
operations.  Testing  shall  be  accomplished  in  realistic  operational  conditions, 
when  feasible  and  essential  to  the  evaluation.  Responsibility  for  managing 
the  practical  aspects  of  each  JT&E  will  be  delegated  to  a  specific  DoD 
Component,  and  supported  by  forces  and  material  from  participating  Components. 

As  the  proponent  for  joint  procedures  and  interoperability  of  deployed 
forces,  the  JCS  have  a  requirement  for  JT&E  results  that  provide  information 
on  joint  doctrine,  tactics,  and  operational  procedures.  Joint  testing 
objectives  will  be  addressed,  when  feasible,  in  conjunction  with  scheduled 
JCS  exercises  to  minimize  resource  inpact  and  provide  economies.  When 
JT&E  and  JCS  exercises  are  integrated,  the  JCS  will  participate,  as  appropriate, 
in  testing  involving  joint  force  interoperability  to  ensure  compatibility 
of  exercise  and  JT&E  objectives. 

a.  The  JCS  shall  annually  coordinate,  for  submission  to  the  DDT&E, 

JT&E  nominations  by  the  Joint  Staff,  the  Military  Services,  and  the  Ccnmanders 
in  Chief  (CINC)  of  the  Unified  and  specified  Commands.  This  does  not 
preclude  direct  nominations  to  the  DOTE  from  the  Military  Services  or 
CINCs  for  JT&E  activities  that  are  inappropriate  for  JCS  consideration 
or  out  of  phase  with  the  JCS  nominations. 

b.  The  list  of  nominations  shall  be  prioritized  for  each  fiscal 
year.  To  the  extent  feasible,  it  shall  identify  the  participating  Military 
Services,  identify  tests  with  potential  for  integration  with  JCS  exercises, 
and  reconroend  a  lead  Service  or  CINC  to  conduct  the  JT&E. 

c.  Control  and  OSD  sponsorship  of  JT&E  will  be  exercised  by  the 
DOTE.  The  DOTE,  in  coordination  with  the  JCS,  will  task  the  selected 
lead  Service  or,  through  the  JCS,  the  selected  CINC  to  conduct  the  test, 
incorporate  the  test  into  joint  exercises,  as  appropriate,  appoint  a 
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Joint  Test  Director,  develop  the  test  plans,  and  provide  reports,  as 
required. 

d.  The  Military  Services,  CINCs  (if  appropriate),  and  the  Joint 
Staff  shall  participate  in  or  monitor  the  JT&E  definition  and  test  design 
efforts,  and  coordinate  the  results  of  these  before  the  ocrmitroent  of 


resources. 


JOINT  TEST  &•  EVALUATION  (JT&E) 
THIRD-LEVEL  FUNCTIONAL  BLOCK  DIAGRAM  (4.5) 


JOINT  TEST  AND  EVALUATION  (4.5) 
RAS  -  SUMMARY  DATA  FORM 


CONVERT  IXI  TO  DEFINITIVE  FIGURES  IN  ACTUAL  PROGRAM  RA 
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RESOURCES  AND  FACILITIES  MANAGEMENT  (4.6) 


POLICY 

o  The  Major  Range  and  Test  Facility  Base  (MKTFB)  is  a  national 
asset  which  shall  be  sized,  operated,  and  maintained  primarily  for  DoD 
test  and  evaluation  support  missions,  but  also  be  available  to  all  users 
having  a  valid  requirement  for  its  capabilities. 

o  Scheduling  of  test  resources  shall  be  based  on  a  priority  system 
that  gives  equitable  consideration  to  all  DoD  Components. 

o  Major  MKTFB  test  and  evaluation  support  capabilities  shall  be 
based  on  a  combination  of  user  requirements  and  the  mission  of  the  activity 
and  shall  not  be  duplicated  elsewhere  within  the  DoD. 

o  When  a  test  requires  the  support  of  more  than  one  MKTFB  activity, 
a  lead  MKTFB  activity  concept  shall  be  used. 

o  The  Major  Range  and  Test  Facility  Ccnmittee  shall  act  as  a  permanent 
organization  to  coordinate  planning  and  actions  with  respect  to  the  MKTFB. 
PROCESS  DESCRIPTION 

o  Test  and  evaluation  support  activities  within  the  MKTFB  are  defined 
by  an  iterative  process  initiated  by  a  program  manager  having  a  requirement 
for  such  support.  The  program’s  T&E  needs  are  stated  in  a  program  introduction 
document,  which  is  provided  by  the  program  manager  to  the  test  range  or 
ranges  best  suited  to  support  the  program  from  the  standpoint  of  instru¬ 
mentation,  personnel,  airspace  and  environmental  considerations,  etc. 

This  document  supplies  a  greater  degree  of  detail  on  test  resource  require¬ 
ments  than  that  outlined  in  the  program  TEMP  (required  by  DoD  Directive 
5000.3).  The  test  requirements  are  reviewed  by  the  range,  which  returns 
to  the  program  manager  an  assessment  of  the  range's  capabilities  to  provide 


the  desired  support,  including  perhaps  a  list  of  constraints  and  deficiencies, 
such  as  additional  instrumentation  needed  to  support  the  test  but  not 
presently  available.  The  program  manager  evaluates  the  range  capability 
assessment  and  supplies  the  range  with  a  restatement  of  support  needs, 
possibly  revised  to  accomodate  any  significant  restrictions  imposed  by 
limitations  on  range  capability.  (Alternatively,  the  program  manager 
may  agree  to  provide  for  enhancements  to  range  resources  to  overcame  the 
restrictions,  e.g.,  a  better  instrumentation  radar  or  an  improved  telemetry 
system.)  This  interchange  is  repeated  until  it  converges  to  an  agreement 
on  the  type,  scope,  schedule  and  funding  arrangements  for  the  specified 
test  support. 

o  According  to  the  DcD  Uniform  Funding  Policy  for  MRXFB  elements, 
benefitting  programs  are  obliged  to  reimburse  test  ranges  for  the  "direct" 
or  "user"  costs  of  test  support  provided. 

o  In  addition,  each  range  receives  "indirect"  or  " institutional" 
funding  from  its  parent  Service  to  support  improvement  and  modernization 
efforts  (general  maintenance  and  repair,  acquisition  of  new  technology, 
etc.)  and  to  cover  expenses  not  directly  attributable  to  the  requirements 
of  any  particular  program  user.  Such  funding  is  monitored  from  a  joint 
Service  perspective  by  the  DDT&E  to  insure  that  no  unwarranted  duplication 
of  range  resources  is  developed,  while  at  the  same  time  deficiencies  in 
MKTFB  capability  are  adequately  addressed. 


DEFENSE  TEST  AND  EVALUATION  -  RES.  &  FAC.  MGT. 
THIRD-LEVEL  FUNCTIONAL  BLOCK  DIAGRAM  (4.6) 
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DEFENSE  SYSTEM  ACQUISITION  REVIEW  COUNCIL  (DSARC)  (4.7) 


POLICY 

o  Normally,  the  DAE  will  be  assisted  by  the  DSARC  at  Milestones  I, 

II,  and  III*.  Program  reviews  may  be  called  at  any  time  during  the  process 
with  OSD  staff  maintaining  continuous  visibility  over  matters  such  as 
costs,  supportability,  test  and  evaluation. 

o  OSD  and  component  level  milestone  reviews  shall  be  scheduled 
with  reasonable  proximity. 

o  Goals  and  thresholds  for  cost,  schedule,  performance,  and  support- 
ability  shall  be  documented  in  program  documents  (JMSNS,  SCP,  DCP,  IPS 
and  TEMPS)  and  reaffirmed  in  Secretary  of  Defense  Decision  Memorandum 
(SDCM) . 

o  Milestone  I  is  based  on  system  Concept  Papers  that  will  identify 
program  alternatives  with  specific  goals  and  objectives  to  be  met  and 
reviewed  at  Milestone  II.  A  TEMP  is  required  prior  to  Milestone  I. 

o  At  Milestone  II  firm  design- to-costs  requirements  shall  be  established. 
Program  accomplishments  shall  be  evaluated  against  cost,  schedule,  performance 
and  supportability  goals  and  thresholds. 

♦Milestone  III  delegated  to  DcD  components  for  most  programs. 

PROCESS  DESCRIPTION 

The  policy,  objectives,  resources  and  control  procedures  to  be  included 
in  Milestone  I,  II,  and  III  reviews  are  specified  in  JMSNS,  DCP,  IPS  and 
TEMP  documents.  The  Under  Secretary  of  Defense  for  Research  and  Engineering 
is  the  Acquisition  Executive  and  is  Chairman  of  the  DSARC.  Detailed 
assessment  reports  and  briefings  are  the  primary  inputs  reviewed  by  the 
council,  whose  basic  outputs  are  reooninendations  for  Secretary  of  Defense 
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Decision  Memorandum  (SDOM) .  Reaffirmation  of  mission  needs  and  stated 
advanced  development  courses  of  actions,  including  management  constraints, 
are  established  at  Milestone  I-  Performance,  cost  and  schedule  thresholds 
are  def initized  and  a  decision  rendered  on  full  scale  development  at 
Milestone  II.  Approved  of  the  system  for  production  is  accomplished  at 
Milestone  III.  Usually  this  is  a  most  difficult  stage  for  T&E  with  limited 
operational  test  data  being  a  primary  consideration.  Therefore,  incremental 
or  conditional  decisions  are  frequently  rendered  for  transition  of  programs 
from  development  to  production  with  subsequent  reviews  either  by  DEfT&E 
or  the  DSARC. 
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TEST  PLANNING  AND  MONITORING 


POLICY 

o  DoD  components  will  structure  an  effective  test  program  and  provide 
an  evaluation  of  the  planned  development  and  operational  test  program  at 
the  major  program  milestones. 

o  DDT&E  will  monitor  testing  progress  and  report  his  independent 
assessment  to  DAE  and  the  Secretary  of  Defense  as  required. 

o  Because  of  the  need  to  reduce  acquisition  time,  the  maximum  amount 
of  test  inf om*.** ion  obtainable  must  be  collected  as  early  as  possible  in 
the  acquisition  cycle. 

o  Adequate  and  early  funding  of  test  hardware  will  be  closely  examined 
prior  to  and  at  milestone,  program,  and  budget  reviews. 

o  Sufficient  resources  shall  be  provided  for  early  initiation  and 
timely  ccnpletion  of  T&E.  Early  and  imaginative  planning  will  be  required 
and  should  be  reflected  in  the  Test  and  Evaluation  Master  Plan  (TEMP) . 

This  shall  be  the  primary  document  utilized  in  the  test  planning  and 
implementation  process. 

PROCESS  DESCRIPTION 

The  TEMP  is  the  primary  document  utilized  to  guide  and  control  T&E. 

Basic  to  successful  utilization  of  this  document  is  development  and  preliminary 
approval  early  in  the  program  (prior  to  Milestone  I)  with  iterative  changes 
encouraged  throughout  the  acquisition  to  accommodate  refinement  of  data 
elements.  Such  an  approach  establishes  achievable  stepped  demonstrations. 

It  does  not  mean,  as  same  would  imply,  that  critical  requirements  and 
thresholds  are  rubberized.  Basic  contents  of  the  TEMP  include: 
o  Mission  and  System  Description; 
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o  Required  operational  and  technical  characteristics; 
o  Critical  T&£  issues; 
o  Management  responsibilities; 
o  DT&E/OT&E  expected  or  actual  results; 
o  Production  acceptance  T&E; 

o  Test  articles  and  special  support  requirements. 

Perhaps  the  most  important  part  of  this  process  is  the  implementation 
procedures  specified  in  hiearchical  test  plans.  For  example,  it  is  essential 
to  have  critical  data  specified  in  higher  level  plans  included  in  procedures 
utilized  at  the  extremities  of  the  test  organization  e.g.,  the  knee  cards 
of  pilots  flying  the  aircraft  in  the  case  of  the  B-l  program. 


TEST  PLANNING  AND  MONITORING/DSARC 
THIRD-LEVEL  FUNCTIONAL  DIAGRAM  (4.7  &  4.8) 


YEARS  TO  ACCOMPLISH 


I 


33 


DEVELOPMENT  AND  OPERATIONAL  TEST  (4.9) 


POLICY 

o  The  head  of  each  DcO  component  shall  designate  a  single  major 
field  agency,  separate  and  distinct  frcm  the  material  developing/processing 
ccmnands  and  user  representative  commands,  to  be  responsible  for  the 
conduct  of  operational  test  and  evaluation  (OT&E) . 

o  DT&E  will  be  acccmplished  by  DcO  Components  to  assist  the  engineering 
design  and  development  process  and  to  verify  attainment  of  technical 
performance  specifications  and  objectives.  Such  T&E  shall  include  participa¬ 
tion  of  the  DoD  component's  independent  test  activity  and  support  decisions 
required  at  major  program  milestones. 

o  OT&E  shall  be  conducted  by  a  DoD  component's  independent  test 
activity  to  determine  the  operational  effectiveness  and  suitability 
characteristics  of  the  system  in  the  acquisition  cycle.  An  independent 
assessment  shall  be  issued  by  these  activities. 

o  Combining  development  and  operational  testing  is  permitted  under 
the  special  provision  of  DoD  policy,  provided  credibility  of  independent 
OT&E  is  maintained,  an  acquisition  approach  is  established  early  and 
approved  by  SECDEF,  and  a  clear  testing  methodology  has  been  established 
in  an  approved  TEMP. 

o  Production  Acceptance  Test  and  Evaluation  (PAT&E)  shall  be  validated 
by  the  DoD  components  as  part  of  their  acquisition  responsibilities. 

PROCESS  DESCRIPTION 

The  two  principal  types  of  test  and  evaluation  conducted  in  the 


acquisition  process  are  Development  Test  and  Evaluation  (DT&E)  and 
Operational  Test  and  Evaluation  (OT&E) .  DT&E  is  conducted  by  or  under 
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the  supervision  of  the  development  agency  to  evaluate  technical  performance 
of  prototype  equipment.  This  testing  is  generally  conducted  by  engineers 
and  technicians — either  contractor  or  government — in  carefully  controlled 
conditions.  OT&E,  on  the  other  hand,  is  conducted  exclusively  by  military 
personnel  to  determine  the  degree  to  which  new  equipment  fulfills  military 
operational  requirements.  It  is,  as  a  rule,  conducted  under  conditions 
that  duplicate  as  closely  as  possible  the  environment  in  which  the  equipment 
is  expected  to  perform  when  deployed. 

These  assessments  serve  important  functions  in  the  acquisition  process. 
OT&E  assists  in  the  actual  design  and  development  of  a  system  in  which 
initial  designs  are  converted  to  hardware.  An  iterative  process  of  test, 
note  deficiencies,  alter  design,  modify  hardware — "cut  and  try" — provides 
the  necessary  feedback  for  an  orderly  progression  frcm  initial  design 
through  engineering  model  stages  to  production  prototype.  Additionally, 

OT&E  provides  information  on  the  progress  of  new  system  development. 

The  progress  is  ascertained  by  comparing  measured  system  performance 
with  a  set  of  technical  goals  and  objectives  for  the  program.  A  principal 
contribution  of  OT&E,  especially  prior  to  full-scale  engineering  development 
phases,  is  the  assessment  of  alternative  system  concepts  and  technical 
approaches. 

OT&E,  like  OT&E,  also  provides  essential  information  for  decision¬ 
making  by  canparing  system  operational  performance  with  operational 
objectives.  Since  OT&E  conducted  before  system  production  involves  testing 
of  prototypes — often  competing  prototypes — test  results  must  be  extrapolated 
to  predict  the  operational  performance  and  suitability  of  the  final  system. 

Combined  OT&E  and  OT&E  are  often  conducted,  especially  in  the  development 
of  large,  expensive  systems  or  systems  which  will  have  a  small  number 
produced  and  fielded. 
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Production  Acceptance  Test  and  Evaluation  (PAT&E)  is  the  third  category 
of  testing,  conducted  on  production  items  to  demonstrate  that  they  meet 
contract  specifications.  PAT&E  performing  agencies  and  methods  are  more 
varied  than  those  conducting  pre-production  T&E.  Developing  agencies 
sometimes  are  responsible  for  the  conduct  of  PAT&E.  Often  the  Defense 
Contract  Administrative  Service  or  the  Service  Plant  Representative  (e.g., 
NAVPRO  or  APPRO)  will  conduct  PAT&E.  In  the  Navy,  acceptance  testing  of 
ships  and  aircraft  is  the  responsibility  of  the  Beard  of  Inspection  and 
Survey. 

Each  Service  maintains  a  major  field  agency,  separate  and  distinct 
from  both  the  developing  or  procuring  activity  and  the  eventual  user 
activity,  to  be  responsible  for  the  conduct  of  OT&E  and  the  monitoring 
of  OT&E.  Each  such  agency  is  required  to  report  the  results  of  independent 
OT&E  (normally  by  Independent  Evaluation  Report,  IER)  directly  to  the 
Service  Chief  and  to  the  Director  Defense  Test  and  Evaluation  for  major 
weapon  systems  requiring  DSARC  level  reviews.  The  Services'  independent 
OT&E  agencies  are  as  follows: 

Army.  U.S.  Army  Operational  Test  and  Evaluation  Agency  (OTEA) , 

5600  Columbia  Pike,  Falls  Church,  VA  22041 

Navy.  Ccmmander  Operational  Test  and  Evaluation  Force,  (COMOPTEVFOR) , 
Norfolk,  VA  23511. 

Air  Force.  Air  Force  Test  and  Evaluation  Center  (AFTEC) ,  Kirtland 
Air  Force  Base,  NM  87117. 

The  foregoing  organizations  were  established  by  the  Services  to 
fulfill  the  "independent  OT&E"  requirements  of  DoD  policy.  Each  Service 
has  other  activities  that  perform  testing  functions,  generally  within 
its  materiel  development  and  acquisition  structure.  These  activities 


are  configured  and  staffed  for  technical,  that  is,  development,  test  and 
evaluation.  These  activities  are  normally  specified  for  particular  test 
support  in  a  program's  charter  or  char  ter-  implement  ing  documentation 
(e.g.,  the  Test  and  Evaluation  Master  Plan  (TEMP) )  to  provide  test  and/or 
evaluation  support  either  independently  or  as  monitor  agency  for  contractor 
DT&E  efforts. 


accomplish 


DEVELOPMENTAL  &  OPERATIONAL  TESTING  (4.9) 
DESIGN  SYNTHESIS  -  DATA  FORM 


SPfcCIHC  VAlUfcS 


DEVELOPMENTAL  &  OPERATIONAL  TESTING  (4.9) 
TRADE  OFF  ANALYSIS  -  DATA  FORM 


DEFENSE  ACQUISITION  (4.10) 


OUCY 

o  Improved  readiness  and  sustainability  are  primary  objectives  of 
die  acquisition  process  and  operational  suitability  of  deployed  weapon 
systems  is  an  objective  of  equal  importance  with  operational  effectiveness. 

o  An  acquisition  strategy  will  be  developed  at  the  inception  of 
sach  major  acquisition. 

o  Minimizing  the  time  it  takes  to  acquire  material  and  facilities 
to  satisfy  military  needs  shall  be  a  primary  goal  in  the  development  of 
an  acquisition  strategy. 

o  Goals  and  thresholds  for  cost,  schedule,  performance,  and  support- 
ability  shall  be  documented  in  program  documents  (JMSNS,  SCP,  DCP,  IPS , 

&  TEMPS)  and  reaffirmed  in  Secretary  of  Defense  Decision  Memorandum  (SDDM) . 

o  Milestone  I  is  based  on  System  Concept  Papers  that  will  identify 
program  alternatives  with  specific  goals  and  objectives  to  be  met  and 
reviewed  at-  Milestone  II.  A  TEMP  is  required  prior  to  Milestone  I. 

o  At  Milestone  II  (program  review  and  decision  for  FSD)  firm  design- 
to-oost  requirements  shall  be  established.  Program  accomplishments  shall 
be  evaluated  against  cost,  schedule,  performance  and  supportability  goals 
and  thresholds.  FSD  decisions  must  be  supported  by  T&E. 

o  Threshold  values  shall  be  proposed  at  Milestones  I  &  II  by  the 
DoD  component  and  approved  by  SECDEF  for  cost,  schedule,  performance, 
and  supportability. 

o  Production  decisions  (Milestone  III)  must  be  based  on  completion 


of  sufficient  operational  test  and  evaluation  by  the  Service  independent 
test  activity  to  determine  operational  effectiveness  and  suitability. 


